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System and Method for Network Packet Reduction 

Field of the Invention 

The present invention relates generally to computer networks and more 
specifically to reducing packets on a network. 

Copyright Notice/Permission 

A portion of the disclosure of this patent document contains material 
which is subject to copyright protection. The copyright owner has no 
objection to the facsimile reproduction by anyone of the patent document 
or the patent disclosure as it appears in the Patent and Trademark Office 
patent file or records, but otherwise reserves all copyright rights 
whatsoever. The following notice applies to the software and data as 
described below and in the drawings hereto: Copyright © 2000 Rutgers, 
The State University of New Jersey, All Rights Reserved. 

Background of the Invention 

Data communication in a computer network involves the exchange of 
data between two or more entities interconnected by communication 
links and subnetworks. These entities are typically software programs 
executing on hardware computer platforms, such as end stations and 
intermediate stations. Examples of an intermediate station may be a 
router or switch which interconnects the communication links and 
subnetworks to enable transmission of data between the end stations. A 
local area network (LAN) is an example of a subnetwork that provides 
relatively short distance communication among the interconnected 
stations; in contrast, a wide area network (WAN) enables long distance 
communication over links provided by public or private 
telecommunications facilities. Communication software executing on 
the end stations correlates and manages data communication with other 
end stations. The stations typically communicate by exchanging discrete 
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packets or frames of data according to predefined protocols. 
Collectively, these hardware and software components comprise a 
communications network. 
Small packets (less than the defauU link MTU of 512 bytes) dominate 
5 Internet traffic patterns today. Wide-area Internet traffic measurements 
show that as many as sixty percent (60%) of the packets transmitted over 
the Internet backbone are less than 44 bytes. Web servers, for example, 
are a leading generator of small packets. All of the TCP ACKs generated 
by the clients have the same destination (the web server). These ACKs 

1 0 are just 40 bytes in size. Another example of small packet traffic is ICP 
(Internet Cache Protocol) queries generated among web proxies. ICP 
queries are about 60 bytes in size. This domination of small packets in 
Internet traffic patterns is expected to get only worse with the ever 
increasing web traffic and with introduction of new applications such as 

15 VoIP (Voiceover-IP apphcations generate small audio samples, about 40 
to 60 byte packets every 40 ms). The network overhead cost due to 
queuing, processing, routing, differential treatment and delivery is the 
same irrespective of the size of the packet. Hence, small packets impose 
a relatively higher overhead on the network. 

20 Therefore, in order to improve overall network performance, there is a 

need to reduce the number of packets in the network. 

Summary of the Invention 
A system of and method for reducing network packet traffic is 
described. Embodiments of the present invention combine small packets 

25 with the same destination into a larger packet, thereby reducing the 
number of packets flowing in the network. Small packets dominate 
Internet traffic patterns today. The overhead due to queuing, processing, 
routing, differential treatment, and delivery is the same irrespective of the 
size of the packet. Hence, small packets impose a relatively higher 

30 overhead on the network. The domination of small packet traffic is 

expected to get only worse with the ever increasing web applications and 
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the introduction of new applications such as Voiceover IP. To mitigate 
the impact of small packets, a novel approach of combining multiple 
small packets going towards the same destination is provided. 
One aspect of the invention is a computerized method of reducing 
5 packets on a network. The method comprises combining multiple 

packets for a destination into an aggregated packet, and separating the 
aggregated packet into the multiple packets at the destination. 

Still other and further embodiments, aspects and advantages of the 
invention will become apparent by reference to the drawings and by 
1 0 reading the following detailed description. 

Brief Description of the Drawings 
Figure 1 is a block diagram of a traditional communication network. 
Figure 2 is a block diagram of a communication network according to 
an example embodiment of the invention. 
15 Figure 3 A is a block diagram of a traditional router. 

Figure 3B is a block diagram of an aggregator, such as the aggregator 
shown in Figure 2, according to an example embodiment of the present 
invention. 

Figure 4 is a block diagram of a splitter, such as the splitter shown in 

20 Figure 2, according to an example embodiment of the present invention. 
Figure 5 is a block diagram showing how metadata is stored when two 
small packets are combined at a router according to an example 
embodiment of the invention. 

Description of the Embodiments 

25 In the following detailed description of the embodiments, reference is 

made to the accompanying drawings which form a part hereof, and in 
which is shown by way of illustration specific embodiments in which the 
invention may be practiced. These embodiments are described in 
sufficient detail to enable those skilled in the art to practice the invention. 

30 It is to be understood that other embodiments may be utilized and that 
structural, logical and electrical changes may be made without departing 
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from the scope of the present inventions. The following detailed 
description is, therefore, not to be taken in a limiting sense, and the scope 
of the present inventions is defined only by the appended claims. 
A system of and method for reducing network packet traffic is 
5 described below. Embodiments of the present invention combine small 
packets with the same destination into a larger packet, thereby reducing 
the number of packets flowing in the network. 

Figure 1 is a block diagram of a traditional communication network 
100. As shown in Figure 1, the traditional communication network 100 
1 0 comprises a plurality of end stations 1 02 and a plurality of intermediate 
stations 104 interconnected by communications links. In the example 
shown in Figure 1, the intermediate links are routers 104. 

Figure 2 is a block diagram of a communication network 200 
according to an example embodiment of the present invention. As shown 
15 in Figure 2, the communication network 200 comprises a plurality of end 
stations 202 and a plurality of intemiediate stations 204. The 
intermediate stations 204 include an aggregator and a splitter which are 
further described below. The aggregator and the splitter function to 
reduce the packets on the communications link between them. The 
20 aggregator combines a small packets into a larger packet so that the 
routers see a smaller number of packets. The splitter separates these 
packets so that the subsequent stations in the network are unaware of the 
packet aggregation. Aggregators, as well as splitters, can be placed at 
intermediate stations or at end stations in the network. The placement of 
25 aggregators and splitters may be varied depending on traffic patterns, 
level of congestion and software application needs. 

In one embodiment, the aggregators as well as the splitters can be the 
end stations in the network. In an alternate embodiment, the aggregators 
and splitters are placed at ingress and egress routers in the network. The 
30 placement of aggregators and splitters may be manually configured or 
may be dynamically assigned according to a signaling mechanism of the 



4 



present invention which is further described below. In another 
embodiment, the aggregated packets on the communications links 
between the aggregator and the splitter are treated as a special class of 
traffic to further enhance network performance. The number of packets 
in the network are reduced by combining small packets multiple times at 
various points in the network. 

In one embodiment, the placement of the aggregator in the network is 
based on the application and is dynamically determined by the receiver of 
the packets, A signaling mechanism of the present invention allows the 
receiver of the packets to place aggregators at points in the network 
where packets can be combined. If the aggregation at that location in the 
network is ineffective, the aggregators can be dynamically or manually 
removed and, if desired, repositioned at another point in the network. In 
other words, if the application ceases to need aggregation, the network 
can remove the aggregators without affecting the normal network 
routing. Thus, the signalling mechanism allows the aggregation of 
packets to be initiated and stopped dynamically. In an alternate 
embodiment, the signaling mechanism can be initiated and stopped by 
manual configure according to a user. 

Figure 3 A is a block diagram of a traditional router 300. As shown in 
Figure 3A, three packets 302 are received by router 300. The router 300 
performs a routing table 304 lookup to determine the output queue 306 
for each one of the packets. All three packets 302 leave router 300 and 
are transmitted across a communication link. 
Figure 3B is a block diagram of an aggregator such as the aggregator 
shown in Figure 2. As used herein, an aggregator refers to any end station or 
intermediate station capable of combining multiple small packets to the same 
destination and sending them as one packet. For example, an aggregator may be 
implemented as a hardware module or a software module in a router. As shown 
in Figure 3B, three packets 3 1 2 are received by router 3 1 0. The router 3 1 0 
performs a routing table 304 lookup to determine the destination of the packets 
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302. However, unlike the traditional router shown in Figure 3 A, the router 3 1 0 
comprises an aggregation module 315. In this example, the three packets 302 
have the same destination. The aggregation module 315 combines the three 
packets 312 into a single packet 318 referred to herein as an "aggregated packet." 
5 The aggregated packet 3 1 8 is placed in the output queue 3 1 6 for the destination 
and unlike the traditional router, only the single aggregated packet leaves the 
router 310 and is transmitted across the communication link. 

Once such an aggregated packet reaches the destination, all the original 
packets can be regenerated. The node that separates aggregated packets is 

10 referred to herein as a "splitter." Figure 4 is a block diagram of a splitter 400. As 
used herein, a splitter refers to any end station or intermediate station capable of 
separating an aggregated packets into the original multiple packets. For 
example, a splitter may be implemented as a hardware module or a software 
module in a router. As shown in Figure 4, an aggregated packet 402 is received 

15 by router 400. The router 400 performs a routing table 404 lookup to determine 
the destination of the packet 402. However, unlike the traditional router shown 
in Figure 3 A, the router 400 comprises a splitter module 406. The splitter 
module 400 separates the aggregated packet 406 into the original multiple 
packets 408. In one embodiment, the multiple packets 408 are placed in the 

20 output queue 4 1 0 for router 400. 

Aggregation as well as splitting can be performed either at layer 2 (MPLS) 
or at layer 3 (network layer) of the protocol stack. Aggregators at layer 2 require 
a hardware implementation. In an example embodiment, aggregators and 
splitters are implemented using transformer tunnels. Transformer tunnels are 

25 described in more detail in Predeep Sudame and B, R. Bardinath, "Transformer 
tunnels: A framework for providing route-specific adaptions," in Proceedings of 
the VSENIX Annual Technical Conference, June 1998, pp. 191-200, which is 
herein incorporated by reference. However, embodiments of the present 
invention are not limited to being implemented with transformer tunnels and 

30 alternate embodiments are contemplated. To control the number of routing table 
entries at an aggregator node, the splitter can be a node other than the final 
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destination. This way, packets belonging to multiple destinations (for example, 
a subnet) can be combined using a single tunnel. Aggregators may also be 
dynamically placed in the network so that the bottleneck links lie between the 
splitters and the aggregators. 
5 A computerized method of reducing packets on a network according an 

example embodiment of the invention comprises combining multiple packets for 
a destination into an aggregated packet, transmitting the single packet to the 
destination, and separating the aggregated packet into the multiple packets at the 
destination. In one embodiment, an aggregation function combines the multiple 

1 0 packets for a destination into an aggregated packet using a mechanism similar to 
TCP delayed ACKs. Every small packet is delayed by a small amount of time. 
The process of combining small packets continues until either m (a parameter to 
the aggregation function described below) packets have been combined, or a 
timeout occurs. If no other small packets arrive before the timeout, the delayed 

1 5 packet is sent as is. This aggregation mechanism is different from IP reassembly 
(perfomied after IP fragmentation). An aggregation function combines small 
packets, along with their IP headers, to get a larger packet. The receiver 
regenerates all the original packets when it receives such an aggregated packet. 
Thus, this mechanism can be used even for protocols that honor message 

20 boundaries (e.g., UDP). 

In one embodiment, for the receiver to be able to regenerate the original 
packets, some information is added (metadata) to the aggregated packet. Figure 
5 is a block diagram showing how metadata is stored when two small packets are 
combined at a router. The generated larger packet has a new protocol number 

25 (77) in the IP header, (The original protocol therefore has to be stored as part of 
the metadata, along with the original destination of the packet, if it is different 
than the address of the splitter node.) The receiver of this packet handles the 
packet to the protocol handler for this new protocol. This handler is responsible 
for removing all the metadata and restoring the packets. 

30 As small packets are delayed for some time in the hope that they can be 

combined with other small packets, the delayed packets see increased latency. 
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The latency depends on the packet inter-arrival rate, and the timeout set by the 
applications. In the best case, when m packets are combined, the timeout does 
not come into picture, because the buffers are flushed immediately when the 

packets are combined. 

An example embodiment of an aggregation function as described above is 
provided below. The aggregation function is provided for illustrative purposes 
only. 



Parameters: 

10 timeout, m: max packets to be combined 

Initialization code: 

delayedjjacket = null\ 

15 aggregate (p) { 

if(p is large) { 

cancel timer; send delayedj)acket; 
delayedj3acket=/2M//; 

} 

20 send p; return PACKET_SENT; 

} 

if (delayed_packet exists) { 

if (p and delayed_packet can be combined) { 

delayed jjacket = combine (delayed_packet, p); 

25 if (m packets have been combined) { 

cancel timer; send delayed_packet; 
delayed_packet = null; 

} 

return PACKET_SENT; 

30 } else { 

cancel timer; send delayed_packet; 
set timer; delayed_packet = p; 
return PACKET_SENT; 

) 

35 ) else { 

set timer; delayed jjacket =p; 
return PACKET_SENT; 

} 

} 

40 Timer expires: 

send delayedjjacket; delayed_packet = null; 



8 



wo 00/72532 



PCT/USOO/14381 



Given that there is sufficient evidence of small packet traffic in the 
Internet, there are a number of benefits to be obtained by packet-level 
aggregation. First, most router implementations (such as FreeBSD, Linux, and 
many Cisco routers) are packet limited rather than byte limited. They place a 
5 limit on the number of packets that the router can queue irrespective of the size 
of each packet. For reasons of efficiency, routers store packet headers (whose 
sizes are independent of the packet sizes) in fast memory, and the packet payload 
in slow memory. The fast memory being expensive decides the queue size. 
Every packet, irrespective of the size, consumes one slot in the queue. 
10 Combining small packets into a larger packet reduces the number of packets seen 
at routers, and hence helps absorb bursts (reduce losses) for a given level of 
congestion. 

Second, every packet at a router requires a routing table lookup and incurs 
the same performance cost irrespective of the size of the packet. Combining 

1 5 small packets at a router reduces the number of packets seen by all the 

subsequent routers. Moreover, to implement a specific per-hop behavior (as in 
differential services), routers have to give preferential treatment to every packet 
of a particular class. By combining small packets belonging to the same class, 
these overheads can be reduced. 

20 Third, every packet requires a link-layer header. For example, every 

ethemet packet has 1 8 bytes overhead (2 bytes for type, 6 bytes for source 
address, 6 bytes for destination address, 4 bytes for CRC). Moreover, an 
ethemet frame also has 12 bytes of frame spacing and 8 bytes of preamble. Also, 
packets smaller than 46 bytes have to be padded with additional bytes to make 

25 the frame size 64 bytes. Therefore, by combining two 40 bytes packets into a 
larger packet, 44 bytes are saved. The gains increase further if more than two 
packets are combined into a larger packet. 

For gigabit ethemet, the minimum frame size is 512 bytes, and hence even 
bigger savings are obtained by combining multiple small packets into a single 

30 larger packet. Packet bursting technique reduces the padding overheads by 
sending back-to-back frames, however, the first frame still incurs the padding 

9 
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overhead and two consecutive frames must have the required inter-frame 
spacing. 

Fourth, on Unks such as Ethernets and wireless LANs, terminals need to 
contend for the channel prior to sending a packet. Reducing the number of 
5 packets reduces the contention cost and increases channel utilization. 

Finally, aggregation reduces interrupt overheads. Incoming packets 
generate an interrupt at the receiver. On a modem RISC processor, each 
interrupt breaks the CPU pipeline, interferes with the cache, requires a change to 
the memory management context, and forces a substantial number of CPU 
1 0 registers to be saved. An /i-fold reduction in packets thus reduces the interrupt 
and packet overhead by a factor of 

It is to be understood that the above description is intended to be 
illustrative, and not restrictive. Many other embodiments will be apparent to 
those of skill in the art upon reviewing the above description. The scope of the 
15 invention should, therefore, be determined with reference to the appended 

claims, along with the full scope of equivalents to which such claims are entitled. 
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What is claimed is: 

1 . A computerized method of reducing packets on a network, the method 
comprising: 

5 combining multiple packets for a destination into an aggregated packet; 

transmitting the single packet to the destination; 
separating the aggregated packet into the multiple packets at the 
destination. 

10 2. The method of claim 1 , wherein a predetemiined number of the multiple 
packets are combined into the aggregated packet. 

3. The method of claim 1 , wherein the multiple packets received over a 

predetermined period of time are combined into the aggregated packet. 

15 

4. The method of claim 1 wherein the combining and splitting is 
implemented in hardware. 

5. The method of claim 1 wherein the combining and splitting is implement 
20 in software. 

6. The method of claim 1 wherein the destination is an end station, 

7. The method of claim 1 wherein the destination is an intermediate station. 

25 

8. The method of claim 1 wherein the combining is performed at a 
dynamically designated network station. 

9. The method of claim 1 wherein the combining is performed at a 
30 user-designated network station. 

11 
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1 0. A network router comprising: 

a routing table to identify a destination for a plurality of packets; and 
an aggregator module to combine at least two of the packets with a same 
destination into an aggregate packet. 

5 

1 1 . The network router of claim 1 0, wherein a predetermined number of the 
multiple packets are combined into the aggregated packet. 

12. The network router of claim 10, wherein the multiple packets received 

1 0 over a predetermined period of time are combined into the aggregated packet. 

13. The network router of claim 1 0 wherein the aggregator module is a 
hardware module. 

15 14. The network router of claim 1 0 wherein the aggregator module is a 
software module. 

15. A network router comprising : 

a routing table to identify a destination for an aggregated packet; and 
20 a splitter module to regenerate a plurality of original packets from a second 

aggregated packet. 

16. The network router of claim 15 wherein the splitter module is a hardware 
module. 

25 

1 1. The network router of claim 1 5 wherein the splitter module is a software 
module. 

1 8. A computerized system comprising: 
30 an aggregator module to combine multiple packets with a same destination 

into a first aggregate packet; and 
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a splitter module to regenerate a plurality of original packets from a second 
aggregated packet. 

19. The computerized system of claim 18, further comprising a signaling 
5 mechanism to dynamically establish an aggregator module. 

20. A computer readable medium containing executable program instructions 
combining network packets, the executable instructions comprising program 
instructions for: 

10 combining small packets for a same destination into a single larger packet; 

adding meta data to the aggregated packet; and 
sending the aggregated packet to a receiver. 

2 1 . The computer readable medium of claim 20 wherein the combining of 

15 small packets into a single larger packet continues until a predetermined number 
of small packets have been combined into the larger packet. 

22. The computer readable medium of claim 20 wherein the combining of 
small packets continues until a predetermined period of time expires. 
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System and Method for Network Packet Reduction 

5 Field of the Invention 

The present invention relates generally to computer networks and more 
specifically to reducing packets on a network. 

Copyright Notice/Permission 
A portion of the disclosure of this patent document contains material 

1 0 which is subject to copyright protection. The copyright owner has no 

objection to the facsimile reproduction by anyone of the patent document 
or the patent disclosure as it appears in the Patent and Trademark Office 
patent file or records, but otherwise reserves all copyright rights 
whatsoever. The following notice applies to the software and data as 

15 described below and in the drawings hereto: Copyright © 2000 Rutgers, 
The State University of New Jersey, All Rights Reserved. 

Background of the Invention 
Data communication in a computer network involves the exchange of 
data between two or more entities interconnected by communication 

20 links and subnetworks. These entities are typically software programs 
executing on hardware computer platforms, such as end stations and 
intermediate stations. Examples of an intermediate station may be a 
router or switch which interconnects the communication links and 
subnetworks to enable transmission of data between the end stations. A 

25 local area network (LAN) is an example of a subnetwork that provides 
relatively short distance communication among the interconnected 
stations; in contrast, a wide area network (WAN) enables long distance 
communication over links provided by public or private 
telecommunications facilities. Communication software executing on 

30 the end stations correlates and manages data communication with other 
end stations. The stations typically communicate by exchanging discrete 
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packets or frames of data according to predefined protocols. 
Collectively, these hardware and software components comprise a 
communications network. 
Small packets (less than the default link MTU of 512 bytes) dominate 
5 Internet traffic patterns today. Wide-area Internet traffic measurements 
show that as many as sixty percent (60%) of the packets transmitted over 
the Internet backbone are less than 44 bytes. Web servers, for example, 
are a leading generator of small packets. All of the TCP ACKs generated 
by the clients have the same destination (the web server). These ACKs 

1 0 are just 40 bytes in size. Another example of small packet traffic is ICP 
(Internet Cache Protocol) queries generated among web proxies. ICP 
queries are about 60 bytes in size. This domination of small packets in 
Internet traffic patterns is expected to get only worse with the ever 
increasing web traffic and with introduction of new applications such as 

15 VoIP (Voiceover-IP applications generate small audio samples, about 40 
to 60 byte packets every 40 ms). The network overhead cost due to 
queuing, processing, routing, differential treatment and delivery is the 
same irrespective of the size of the packet. Hence, small packets impose 
a relatively higher overhead on the network. 

20 Therefore, in order to improve overall network performance, there is a 

need to reduce the number of packets in the network. 

Summary of the Invention 
A system of and method for reducing network packet traffic is 
described. Embodiments of the present invention combine small packets 

25 with the same destination into a larger packet, thereby reducing the 
number of packets flowing in the network. Small packets dominate 
Internet traffic patterns today. The overhead due to queuing, processing, 
routing, differential treatment, and delivery is the same irrespective of the 
size of the packet. Hence, small packets impose a relatively higher 

30 overhead on the network. The domination of small packet traffic is 

expected to get only worse with the ever increasing web applications and 
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the introduction of new applications such as Voiceover IP. To mitigate 
the impact of small packets, a novel approach of combining multiple 
small packets going towards the same destination is provided. 
One aspect of the invention is a computerized method of reducing 
5 packets on a network. The method comprises combining multiple 

packets for a destination into an aggregated packet, and separating the 
aggregated packet into the multiple packets at the destination. 

Still other and further embodiments, aspects and advantages of the 
invention will become apparent by reference to the drawings and by 
1 0 reading the following detailed description. 

Brief Description of the Drawings 
Figure 1 is a block diagram of a traditional communication network. 
Figure 2 is a block diagram of a communication network according to 
an example embodiment of the invention. 
15 Figure 3 A is a block diagram of a traditional router. 

Figure 3B is a block diagram of an aggregator, such as the aggregator 
shown in Figure 2, according to an example embodiment of the present 
invention. 

Figure 4 is a block diagram of a splitter, such as the splitter shown in 
20 Figure 2, according to an example embodiment of the present invention. 

Figure 5 is a block diagram showing how metadata is stored when two 
small packets are combined at a router according to an example 
embodiment of the invention. 

Description of the Embodiments 

25 In the following detailed description of the embodiments, reference is 

made to the accompanying drawings which form a part hereof, and in 
which is shown by way of illustration specific embodiments in which the 
invention may be practiced. These embodiments are described in 
sufficient detail to enable those skilled in the art to practice the invention. 

30 It is to be understood that other embodiments may be utilized and that 
structural, logical and electrical changes may be made without departing 
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from the scope of the present inventions. The following detailed 
description is, therefore, not to be taken in a limiting sense, and the scope 
of the present inventions is defined only by the appended claims. 
A system of and method for reducing network packet traffic is 
5 described below. Embodiments of the present invention combine small 
packets with the same destination into a larger packet, thereby reducing 
the number of packets flowing in the network. 

Figure 1 is a block diagram of a traditional communication network 
100. As shown in Figure 1, the traditional communication network 100 

1 0 comprises a plurality of end stations 1 02 and a plurality of intermediate 
stations 104 interconnected by communications links. In the example 
shown in Figure 1, the intermediate links are routers 104. 

Figure 2 is a block diagram of a communication network 200 
according to an example embodiment of the present invention. As shown 

15 in Figure 2, the communication network 200 comprises a plurality of end 
stations 202 and a plurality of intermediate stations 204. The 
intermediate stations 204 include an aggregator and a splitter which are 
further described below. The aggregator and the splitter function to 
reduce the packets on the communications link between them. The 

20 aggregator combines a small packets into a larger packet so that the 
routers see a smaller number of packets. The splitter separates these 
packets so that the subsequent stations in the network are unaware of the 
packet aggregation. Aggregators, as well as splitters, can be placed at 
intermediate stations or at end stations in the network. The placement of 

25 aggregators and splitters may be varied depending on traffic patterns, 
level of congestion and software application needs. 

In one embodiment, the aggregators as well as the splitters can be the 
end stations in the network. In an alternate embodiment, the aggregators 
and splitters are placed at ingress and egress routers in the network. The 

30 placement of aggregators and splitters may be manually configured or 
may be dynamically assigned according to a signaling mechanism of the 
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present invention which is further described below. In another 
embodiment, the aggregated packets on the communications links 
between the aggregator and the splitter are treated as a special class of 
traffic to further enhance network perfomiance. The number of packets 
5 in the network are reduced by combining small packets multiple times at 
various points in the network. 

In one embodiment, the placement of the aggregator in the network is 
based on the application and is dynamically determined by the receiver of 
the packets. A signahng mechanism of the present invention allows the 
1 0 receiver of the packets to place aggregators at points in the network 

where packets can be combined. If the aggregation at that location in the 
network is ineffective, the aggregators can be dynamically or manually 
removed and, if desired, repositioned at another point in the network. In 
other words, if the application ceases to need aggregation, the network 
15 can remove the aggregators without affecting the normal network 
routing. Thus, the signalling mechanism allows the aggregation of 
packets to be initiated and stopped dynamically. In an alternate 
embodiment, the signaling mechanism can be initiated and stopped by 
manual configure according to a user. 
20 Figure 3 A is a block diagram of a traditional router 300. As shown in 

Figure 3 A, three packets 302 are received by router 300. The router 300 
performs a routing table 304 lookup to determine the output queue 306 
for each one of the packets. All three packets 302 leave router 300 and 
are transmitted across a communication link. 
25 Figure 3B is a block diagram of an aggregator such as the aggregator 

shown in Figure 2. As used herein, an aggregator refers to any end station or 
intermediate station capable of combining multiple small packets to the same 
destination and sending them as one packet. For example, an aggregator may be 
implemented as a hardware module or a software module in a router. As shown 
30 in Figure 3B, three packets 3 1 2 are received by router 3 1 0. The router 3 1 0 
performs a routing table 304 lookup to determine the destination of the packets 
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302. However, unlike the traditional router shown in Figure 3 A, the router 310 
comprises an aggregation module 315. In this example, the three packets 302 
have the same destination. The aggregation module 315 combines the three 
packets 312 into a single packet 318 referred to herein as an "aggregated packet." 
5 The aggregated packet 3 1 8 is placed in the output queue 3 1 6 for the destination 
and unlike the traditional router, only the single aggregated packet leaves the 
router 310 and is transmitted across the communication link. 

Once such an aggregated packet reaches the destination, all the original 
packets can be regenerated. The node that separates aggregated packets is 

10 referred to herein as a "splitter." Figure 4 is a block diagram of a splitter 400. As 
used herein, a splitter refers to any end station or intermediate station capable of 
separating an aggregated packets into the original multiple packets. For 
example, a splitter may be implemented as a hardware module or a software 
module in a router. As shown in Figure 4, an aggregated packet 402 is received 

1 5 by router 400. The router 400 performs a routing table 404 lookup to determine 
the destination of the packet 402. However, unlike the traditional router shown 
in Figure 3 A, the router 400 comprises a splitter module 406. The splitter 
module 400 separates the aggregated packet 406 into the original multiple 
packets 408. In one embodiment, the multiple packets 408 are placed in the 

20 output queue 4 1 0 for router 400. 

Aggregation as well as splitting can be performed either at layer 2 (MPLS) 
or at layer 3 (network layer) of the protocol stack. Aggregators at layer 2 require 
a hardware implementation. In an example embodiment, aggregators and 
splitters are implemented using transfonner tunnels. Transformer tunnels are 

25 described in more detail in Predeep Sudame and B. R. Bardinath, "Transfonner 
tunnels: A framework for providing route-specific adaptions," in Proceedings of 
the USENIX Annual Technical Conference, June 1998, pp. 191-200, which is 
herein incorporated by reference. However, embodiments of the present 
invention are not limited to being implemented with transformer tunnels and 

30 alternate embodiments are contemplated. To control the number of routing table 
entries at an aggregator node, the splitter can be a node other than the final 
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destination. This way, packets belonging to multiple destinations (for example, 
a subnet) can be combined using a single tunnel. Aggregators may also be 
dynamically placed in the network so that the bottleneck links lie between the 
splitters and the aggregators. 
5 A computerized method of reducing packets on a network according an 

example embodiment of the invention comprises combining multiple packets for 
a destination into an aggregated packet, transmitting the single packet to the 
destination, and separating the aggregated packet into the multiple packets at the 
destination. In one embodiment, an aggregation function combines the multiple 

10 packets for a destination into an aggregated packet using a mechanism similar to 
TCP delayed ACKs. Every small packet is delayed by a small amount of time. 
The process of combining small packets continues until either m (a parameter to 
the aggregation function described below) packets have been combined, or a 
timeout occurs. If no other small packets arrive before the timeout, the delayed 

15 packet is sent as is. This aggregation mechanism is different from IP reassembly 
(performed after IP fragmentation). An aggregation function combines small 
packets, along with their IP headers, to get a larger packet. The receiver 
regenerates all the original packets when it receives such an aggregated packet. 
Thus, this mechanism can be used even for protocols that honor message 

20 boundaries (e.g., UDP), 

In one embodiment, for the receiver to be able to regenerate the original 
packets, some information is added (metadata) to the aggregated packet. Figure 
5 is a block diagram showing how metadata is stored when two small packets are 
combined at a router. The generated larger packet has a new protocol number 

25 (77) in the IP header. (The original protocol therefore has to be stored as part of 
the metadata, along with the original destination of the packet, if it is different 
than the address of the splitter node.) The receiver of this packet handles the 
packet to the protocol handler for this new protocol This handler is responsible 
for removing all the metadata and restoring the packets. 

30 As small packets are delayed for some time in the hope that they can be 

combined with other small packets, the delayed packets see increased latency. 
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The latency depends on the packet inter-arrival rate, and the timeout set by the 
applications. In the best case, when m packets are combined, the timeout does 
not come into picture, because the buffers are flushed immediately when the 
packets are combined. 
5 An example embodiment of an aggregation function as described above is 

provided below. The aggregation function is provided for illustrative purposes 
only. 



Parameters: 

10 timeout, m\ max packets to be combined 

Initialization code: 

delayed jacket = null, 

15 aggregate (p) { 

if (p is large) { 

cancel timer; send delayed_packet; 
delayedj3acket=/aM//; 

} 

20 send p; return PACKET_SENT; 

} 

if (delayed_packet exists) { 

if (p and delayed_packet can be combined) { 

delayed j5acket = combine (delayed_packet, p); 
25 if (m packets have been combined) { 

cancel timer; send delayed_packet; 
delayed__packet = null; 

) 

return PACKET_SENT; 

30 } else { 

cancel timer; send delayed_packet; 
set timer; delayed_packet = p; 
return PACKET_SENT; 
} 

35 } else { 

set timer; delayed_packet = p; 
return PACKET_SENT; 

} 

} 

40 Timer expires: 

send delayed_packet; delayedjsacket = null; 
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Given that there is sufficient evidence of small packet traffic in the 
Internet, there are a number of benefits to be obtained by packet-level 
aggregation. First, most router implementations (such as FreeBSD, Linux, and 
many Cisco routers) are packet limited rather than byte limited. They place a 

5 limit on the number of packets that the router can queue irrespective of the size 
of each packet. For reasons of efficiency, routers store packet headers (whose 
sizes are independent of the packet sizes) in fast memory, and the packet payload 
in slow memory. The fast memory being expensive decides the queue size. 
Every packet, irrespective of the size, consumes one slot in the queue. 

1 0 Combining small packets into a larger packet reduces the number of packets seen 
at routers, and hence helps absorb bursts (reduce losses) for a given level of 
congestion. 

Second, every packet at a router requires a routing table lookup and incurs 
the same performance cost irrespective of the size of the packet. Combining 

1 5 small packets at a router reduces the number of packets seen by all the 

subsequent routers. Moreover, to implement a specific per-hop behavior (as in 
differential services), routers have to give preferential treatment to every packet 
of a particular class. By combining small packets belonging to the same class, 
these overheads can be reduced. 

20 Third, every packet requires a link-layer header. For example, every 

ethemet packet has 18 bytes overhead (2 bytes for type, 6 bytes for source 
address, 6 bytes for destination address, 4 bytes for CRC). Moreover, an 
ethemet frame also has 12 bytes of frame spacing and 8 bytes of preamble. Also, 
packets smaller than 46 bytes have to be padded with additional bytes to make 

25 the frame size 64 bytes. Therefore, by combining two 40 bytes packets into a 
larger packet, 44 bytes are saved. The gains increase further if more than two 
packets are combined into a larger packet. 

For gigabit ethemet, the minimum frame size is 512 bytes, and hence even 
bigger savings are obtained by combining multiple small packets into a single 

30 larger packet. Packet bursting technique reduces the padding overheads by 
sending back-to-back frames, however, the first frame still incurs the padding 
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overhead and two consecutive frames must have the required inter-frame 
spacing. 

Fourth, on links such as Ethernets and wireless LANs, terminals need to 
contend for the channel prior to sending a packet. Reducing the number of 
5 packets reduces the contention cost and increases channel utilization. 

Finally, aggregation reduces interrupt overheads. Incoming packets 
generate an interrupt at the receiver. On a modem RISC processor, each 
interrupt breaks the CPU pipeline, interferes with the cache, requires a change to 
the memory management context, and forces a substantial number of CPU 
10 registers to be saved. An «-fold reduction in packets thus reduces the interrupt 
and packet overhead by a factor of 

It is to be understood that the above description is intended to be 
illustrative, and not restrictive. Many other embodiments will be apparent to 
those of skill in the art upon reviewing the above description. The scope of the 
15 invention should, therefore, be determined with reference to the appended 

claims, along with the full scope of equivalents to which such claims are entitled. 
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What is claimed is: 

1 . A computerized method of reducing packets on a network, the method 
comprising: 

5 combining multiple packets for a destination into an aggregated packet; 

transmitting the single packet to the destination; 
separating the aggregated packet into the multiple packets at the 
destination, 

10 2. The method of claim 1 , wherein a predetermined number of the multiple 
packets are combined into the aggregated packet. 

3. The method of claim 1 , wherein the multiple packets received over a 

predetermined period of time are combined into the aggregated packet. 

15 

4. The method of claim 1 wherein the combining and splitting is 
implemented in hardware. 

5. The method of claim 1 wherein the combining and splitting is implement 
20 in software. 

6. The method of claim 1 wherein the destination is an end station. 

7. The method of claim 1 wherein the destination is an intermediate station. 

25 

8. The method of claim 1 wherein the combining is performed at a 
dynamically designated network station. 

9. The method of claim 1 wherein the combining is performed at a 
30 user-designated network station. 



11 



wo 00/72532 



1 0. A network router comprising: 

a routing table to identify a destination for a plurality of packets; and 
an aggregator module to combine at least two of the packets with a same 
destination into an aggregate packet, 

5 

11. The network router of claim 1 0, wherein a predetermined number of the 
multiple packets are combined into the aggregated packet. 

12. The network router of claim 10, wherein the multiple packets received 
10 over a predetermined period of time are combined into the aggregated packet. 

13. The network router of claim 10 wherein the aggregator module is a 
hardware module. 

15 14. The network router of claim 1 0 wherein the aggregator module is a 
software module. 

15. A network router compri sing : 

a routing table to identify a destination for an aggregated packet; and 
20 a splitter module to regenerate a plurality of original packets from a second 

aggregated packet. 

16. The network router of claim 1 5 wherein the splitter module is a hardware 
module. 

25 

1 7. The network router of claim 1 5 wherein the splitter module is a software 
module. 

18. A computerized system comprising: 

30 an aggregator module to combine multiple packets with a same destination 

into a first aggregate packet; and 
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a splitter module to regenerate a plurality of original packets from a second 
aggregated packet. 

19. The computerized system of claim 1 8, further comprising a signaling 
5 mechanism to dynamically establish an aggregator module. 

20. A computer readable medium containing executable program instructions 
combining network packets, the executable instructions comprising program 
instructions for: 

1 0 combining small packets for a same destination into a single larger packet; 

adding meta data to the aggregated packet; and 
sending the aggregated packet to a receiver. 

2 1 . The computer readable medium of claim 20 wherein the combining of 

15 small packets into a single larger packet continues until a predetermined number 
of small packets have been combined into the larger packet. 

22. The computer readable medium of claim 20 wherein the combining of 
small packets continues until a predetermined period of time expires. 
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